<html>
<head><meta charset="utf-8"><title>Manuel from Germany · new members · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/122652-new-members/index.html">new members</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/122652-new-members/topic/Manuel.20from.20Germany.html">Manuel from Germany</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="225939273"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122652-new%20members/topic/Manuel%20from%20Germany/near/225939273" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Manuel Drehwald <a href="https://rust-lang.github.io/zulip_archive/stream/122652-new-members/topic/Manuel.20from.20Germany.html#225939273">(Feb 11 2021 at 01:50)</a>:</h4>
<p>Hi all, <br>
I'm an undergrad student from Germany. I'm mostly interested in Neural Networks. <br>
Recently I ran into <a href="https://enzyme.mit.edu/">https://enzyme.mit.edu/</a> which calculates gradients based on LLVM-IR.<br>
There is a test repository where it is accessed from Rust, but by manually replacing part of the rustc output with the relevant information. ( e.g. here <a href="https://github.com/tiberiusferreira/oxide-enzyme/blob/master/src/main.rs">https://github.com/tiberiusferreira/oxide-enzyme/blob/master/src/main.rs</a>)<br>
They discussed adding a custom backend which is able to parse an enzyme call (and probably leaving everything else unchanged).</p>
<p>I am entirely new to the compiler topic and just know a few basics from university, like AST and k-lookahead.<br>
This summer I will write my final undergrad thesis and I'm wondering if improving the enzyme integration might be a suitable topic.<br>
The time schedule would be around 300h, so for now I'm trying to find out how complicated that will be and whether there is a good chance<br>
of ending up with a good result.<br>
I would like to know if creating such an mvp would require writing a few hundred or multiple thousand LoC and where to look up how such a work<br>
would be carried out.</p>



<a name="225941498"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122652-new%20members/topic/Manuel%20from%20Germany/near/225941498" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/122652-new-members/topic/Manuel.20from.20Germany.html#225941498">(Feb 11 2021 at 02:26)</a>:</h4>
<p>I don't understand anything on that homepage <span aria-label="laughing" class="emoji emoji-1f606" role="img" title="laughing">:laughing:</span> what is automatic diffentiation? What does it have to do with LLVM IR?</p>



<a name="225942304"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122652-new%20members/topic/Manuel%20from%20Germany/near/225942304" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Manuel Drehwald <a href="https://rust-lang.github.io/zulip_archive/stream/122652-new-members/topic/Manuel.20from.20Germany.html#225942304">(Feb 11 2021 at 02:41)</a>:</h4>
<p><span class="user-mention silent" data-user-id="232545">Joshua Nelson</span> </p>
<p>No worries, this is also new for me. Let me try to explain (hopefully none of the original devs ever has to read that)<br>
Basically you are trying to get the derivative of a function. So when having x^2 as input, you want to get 2x back.<br>
In general it seems like most libraries who support this work by overloading the operators or having some kind of macro-substitution and just storing a lot of information. Then they calculate the derivative by some magic (our 2x) and place it at the location where you tell them.</p>
<p>Regarding enzyme the authors added the ability to llvm to trace a function / code-interval and calculate the derivative based on the llvm-ir.<br>
So that in a higher language you can just say something like:</p>
<p>let x = 5;<br>
_enzyme_start(x);<br>
x *= 2;<br>
x *= 3;<br>
_enzyme_end(x);</p>
<p>let g = _enzyme_gradient(x); // g = 2*3 = 6;</p>
<p>By now, rustc would just complain that _enzyme_x is an undefined function. <br>
A solution would be to tell him to embed the calls into LLVM-IR, since enzyme knows how to handle it.</p>
<p>Please take it with a grain of salt, I'm neither an expert in AD, nor do I know much about enzyme. <span aria-label="sweat smile" class="emoji emoji-1f605" role="img" title="sweat smile">:sweat_smile:</span></p>



<a name="225942437"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122652-new%20members/topic/Manuel%20from%20Germany/near/225942437" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> oliver <a href="https://rust-lang.github.io/zulip_archive/stream/122652-new-members/topic/Manuel.20from.20Germany.html#225942437">(Feb 11 2021 at 02:44)</a>:</h4>
<p>I've been interested in this subject from the PL side since there are a number of DSLs based around these <br>
techniques, my research was in the Stan probabilistic programming language (PPL) which is a way of expressing<br>
needed math and having it compile to C++</p>



<a name="225942644"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122652-new%20members/topic/Manuel%20from%20Germany/near/225942644" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/122652-new-members/topic/Manuel.20from.20Germany.html#225942644">(Feb 11 2021 at 02:48)</a>:</h4>
<p>what is the point though? Why are you trying to find the derivative?</p>



<a name="225942672"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122652-new%20members/topic/Manuel%20from%20Germany/near/225942672" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> oliver <a href="https://rust-lang.github.io/zulip_archive/stream/122652-new-members/topic/Manuel.20from.20Germany.html#225942672">(Feb 11 2021 at 02:48)</a>:</h4>
<p>the derivative is incredibly important for a number of reasons, most generally in min/max identification</p>



<a name="225942914"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122652-new%20members/topic/Manuel%20from%20Germany/near/225942914" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> oliver <a href="https://rust-lang.github.io/zulip_archive/stream/122652-new-members/topic/Manuel.20from.20Germany.html#225942914">(Feb 11 2021 at 02:53)</a>:</h4>
<p>autodiff is just one way the alternative is to define the gradients analytically which may or may not be humanly possible</p>



<a name="225942968"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122652-new%20members/topic/Manuel%20from%20Germany/near/225942968" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> oliver <a href="https://rust-lang.github.io/zulip_archive/stream/122652-new-members/topic/Manuel.20from.20Germany.html#225942968">(Feb 11 2021 at 02:54)</a>:</h4>
<p>analytically is to say how one does their Calc homework in college</p>



<a name="225943002"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122652-new%20members/topic/Manuel%20from%20Germany/near/225943002" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Manuel Drehwald <a href="https://rust-lang.github.io/zulip_archive/stream/122652-new-members/topic/Manuel.20from.20Germany.html#225943002">(Feb 11 2021 at 02:55)</a>:</h4>
<p>What <span class="user-mention" data-user-id="281739">@oliver</span>  says + at least regarding NeuralNetworks people tend to frequently play around with new functions to see which one solves a given task best.<br>
Calculating the derivative for each of those manually can become complex for more complex functions, especially when considering corner cases (e.g. division by zero) and you also need to have it implemented in an efficient way (since you will spend hundreds / thousand of cpu/gpu hours on that).<br>
So the idea is to just have to define the function and let a library take over the rest.<br>
Enzyme claims to be more efficient than other implementations since it is integrated into llvm, allowing more intelligent optimizations.</p>



<a name="225943182"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122652-new%20members/topic/Manuel%20from%20Germany/near/225943182" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Manuel Drehwald <a href="https://rust-lang.github.io/zulip_archive/stream/122652-new-members/topic/Manuel.20from.20Germany.html#225943182">(Feb 11 2021 at 02:59)</a>:</h4>
<p>Here are some other examples. <br>
<a href="https://pytorch.org/docs/stable/autograd.html">https://pytorch.org/docs/stable/autograd.html</a><br>
<a href="https://www.tensorflow.org/api_docs/python/tf/autodiff">https://www.tensorflow.org/api_docs/python/tf/autodiff</a><br>
Both as part of Neural Network Frameworks, but the concept is not limited to that, as pointed out above.</p>



<a name="225943273"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122652-new%20members/topic/Manuel%20from%20Germany/near/225943273" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> oliver <a href="https://rust-lang.github.io/zulip_archive/stream/122652-new-members/topic/Manuel.20from.20Germany.html#225943273">(Feb 11 2021 at 03:00)</a>:</h4>
<p>C++ had a proposal to add autodiff but I can't find it now</p>



<a name="225943286"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122652-new%20members/topic/Manuel%20from%20Germany/near/225943286" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> oliver <a href="https://rust-lang.github.io/zulip_archive/stream/122652-new-members/topic/Manuel.20from.20Germany.html#225943286">(Feb 11 2021 at 03:00)</a>:</h4>
<p>oh: <a href="http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p2072r0.pdf">http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p2072r0.pdf</a></p>



<a name="225943305"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122652-new%20members/topic/Manuel%20from%20Germany/near/225943305" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> oliver <a href="https://rust-lang.github.io/zulip_archive/stream/122652-new-members/topic/Manuel.20from.20Germany.html#225943305">(Feb 11 2021 at 03:01)</a>:</h4>
<p>C++ has _everything_ <span aria-label="laughing" class="emoji emoji-1f606" role="img" title="laughing">:laughing:</span></p>



<a name="225955087"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122652-new%20members/topic/Manuel%20from%20Germany/near/225955087" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> namibj <a href="https://rust-lang.github.io/zulip_archive/stream/122652-new-members/topic/Manuel.20from.20Germany.html#225955087">(Feb 11 2021 at 07:10)</a>:</h4>
<p><span class="user-mention silent" data-user-id="232545">Joshua Nelson</span> <a href="#narrow/stream/122652-new-members/topic/Manuel.20from.20Germany/near/225941498">said</a>:</p>
<blockquote>
<p>I don't understand anything on that homepage <span aria-label="laughing" class="emoji emoji-1f606" role="img" title="laughing">:laughing:</span> what is automatic diffentiation? What does it have to do with LLVM IR?</p>
</blockquote>
<p>The <a href="https://enzyme.mit.edu/getting_started/CallingConvention/">API</a> for user code isn't particularly complex, but the functionality is implemented as an LLVM opt pass.<br>
Ideally it's fed with optimized IR (but without vectorization), followed by another round of optimization (this time with vectorization).</p>
<p>At least in C, it's just hijacking all functions that contain <code>__enzyme_autodiff</code> in their (presumably mangled) name, who's calls take a function pointer, inputs, outputs for the computed gradients, and optional parameter type hints where you just pass one of a few magic globals as an argument directly in front of the argument it should apply to. Quite similar to cli args for e.g. <code>git</code> (where the heuristics sometimes interpret a branch name as a file name or vice-versa).</p>
<p>The <em>Getting Started</em> category I linked to makes no mention of code regions.<br>
Functions seem to work, though.</p>
<p>And the undefined function complains are to be treated like any other C FFI, e.g. with declaring the symbols before use and applying <code>#[no_mangle]</code>.</p>



<a name="225955297"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122652-new%20members/topic/Manuel%20from%20Germany/near/225955297" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> namibj <a href="https://rust-lang.github.io/zulip_archive/stream/122652-new-members/topic/Manuel.20from.20Germany.html#225955297">(Feb 11 2021 at 07:15)</a>:</h4>
<p><span class="user-mention silent" data-user-id="232545">Joshua Nelson</span> <a href="#narrow/stream/122652-new-members/topic/Manuel.20from.20Germany/near/225942644">said</a>:</p>
<blockquote>
<p>what is the point though? Why are you trying to find the derivative?</p>
</blockquote>
<p>AFAIK the predominant use case for such autodiff tools is to run a variant of gradient descent to optimize a non-trivial function expressed as code.<br>
I.e., numerical optimization.</p>



<a name="226273458"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122652-new%20members/topic/Manuel%20from%20Germany/near/226273458" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> oliver <a href="https://rust-lang.github.io/zulip_archive/stream/122652-new-members/topic/Manuel.20from.20Germany.html#226273458">(Feb 13 2021 at 22:53)</a>:</h4>
<p>you might have seen this: <a href="https://github.com/tensorflow/swift#swift-for-tensorflow-in-archive-mode">https://github.com/tensorflow/swift#swift-for-tensorflow-in-archive-mode</a></p>



<a name="226274506"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122652-new%20members/topic/Manuel%20from%20Germany/near/226274506" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> oliver <a href="https://rust-lang.github.io/zulip_archive/stream/122652-new-members/topic/Manuel.20from.20Germany.html#226274506">(Feb 13 2021 at 23:18)</a>:</h4>
<p>why is the Rust bindings to TF so much bigger than for Swift and TF its 3MB v 40MB?</p>



<a name="226274621"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122652-new%20members/topic/Manuel%20from%20Germany/near/226274621" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> oliver <a href="https://rust-lang.github.io/zulip_archive/stream/122652-new-members/topic/Manuel.20from.20Germany.html#226274621">(Feb 13 2021 at 23:21)</a>:</h4>
<p>I thought they were putting lots of work into Swift Tensorflow yet it's a fraction of Rust <span aria-label="laughing" class="emoji emoji-1f606" role="img" title="laughing">:laughing:</span></p>



<a name="226276893"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122652-new%20members/topic/Manuel%20from%20Germany/near/226276893" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Poliorcetics <a href="https://rust-lang.github.io/zulip_archive/stream/122652-new-members/topic/Manuel.20from.20Germany.html#226276893">(Feb 14 2021 at 00:20)</a>:</h4>
<p>They were putting LOTS of work into TF in Swift last I heard, to the point of maintaining an alternate fork of the compiler IIRC</p>



<a name="226320325"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122652-new%20members/topic/Manuel%20from%20Germany/near/226320325" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> oliver <a href="https://rust-lang.github.io/zulip_archive/stream/122652-new-members/topic/Manuel.20from.20Germany.html#226320325">(Feb 14 2021 at 19:23)</a>:</h4>
<p>even some of the contributors didn't seem to know it was being cancelled</p>



<a name="226320383"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122652-new%20members/topic/Manuel%20from%20Germany/near/226320383" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> oliver <a href="https://rust-lang.github.io/zulip_archive/stream/122652-new-members/topic/Manuel.20from.20Germany.html#226320383">(Feb 14 2021 at 19:24)</a>:</h4>
<p><a href="https://forums.swift.org/t/differentiable-programming-for-gradient-based-machine-learning/42147/101">https://forums.swift.org/t/differentiable-programming-for-gradient-based-machine-learning/42147/101</a></p>



<a name="226320399"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122652-new%20members/topic/Manuel%20from%20Germany/near/226320399" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> oliver <a href="https://rust-lang.github.io/zulip_archive/stream/122652-new-members/topic/Manuel.20from.20Germany.html#226320399">(Feb 14 2021 at 19:25)</a>:</h4>
<p><a href="https://twitter.com/TroyAaronHarvey/status/1360718493734215685">https://twitter.com/TroyAaronHarvey/status/1360718493734215685</a></p>
<div class="inline-preview-twitter"><div class="twitter-tweet"><a href="https://twitter.com/TroyAaronHarvey/status/1360718493734215685"><img class="twitter-avatar" src="https://pbs.twimg.com/profile_images/1238536029239304192/pwiLiPPi_normal.jpg"></a><p>We are indebted to the #S4TF team at Google for all their effort... Swift became the 1st language with 1st class differentiable compiler support. We at PassiveLogic continue to invest deeply Swift auto-diff... as we forge new technologies beyond "naive" deep learning.</p><span>- Troy Harvey (@TroyAaronHarvey)</span></div></div>



<a name="226873000"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122652-new%20members/topic/Manuel%20from%20Germany/near/226873000" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Chris Black <a href="https://rust-lang.github.io/zulip_archive/stream/122652-new-members/topic/Manuel.20from.20Germany.html#226873000">(Feb 18 2021 at 20:14)</a>:</h4>
<p><span class="user-mention silent" data-user-id="281739">oliver</span> <a href="#narrow/stream/122652-new-members/topic/Manuel.20from.20Germany/near/226274621">said</a>:</p>
<blockquote>
<p>I thought they were putting lots of work into Swift Tensorflow yet it's a fraction of Rust <span aria-label="laughing" class="emoji emoji-1f606" role="img" title="laughing">:laughing:</span></p>
</blockquote>
<p>The main interest in differentiable functions for Rust would be reinforcement learning, where more robust programming models are need to combine with DL gradients.   S4TF definitely pushed the envelope.  Too bad it got caught in the middle of two tech giants.</p>



<a name="227043274"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122652-new%20members/topic/Manuel%20from%20Germany/near/227043274" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Irene Knapp <a href="https://rust-lang.github.io/zulip_archive/stream/122652-new-members/topic/Manuel.20from.20Germany.html#227043274">(Feb 19 2021 at 23:01)</a>:</h4>
<p>(sorry - inadvertently posted my intro as a reply)</p>



<a name="228661714"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122652-new%20members/topic/Manuel%20from%20Germany/near/228661714" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> oliver <a href="https://rust-lang.github.io/zulip_archive/stream/122652-new-members/topic/Manuel.20from.20Germany.html#228661714">(Mar 03 2021 at 20:24)</a>:</h4>
<p><a href="https://arxiv.org/abs/2102.13243">https://arxiv.org/abs/2102.13243</a></p>



<a name="228662753"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122652-new%20members/topic/Manuel%20from%20Germany/near/228662753" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> oliver <a href="https://rust-lang.github.io/zulip_archive/stream/122652-new-members/topic/Manuel.20from.20Germany.html#228662753">(Mar 03 2021 at 20:31)</a>:</h4>
<p>"To our knowledge, this combination of (1) requesting differ-<br>
entiation within the language, (2) supporting user-defined<br>
types with arbitrary tangent vector types, and (3) performing<br>
the AD code transformation at AOT-compile time was first<br>
explored in Swift for TensorFlow."</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>